Rapid serial visual presentation triage prioritization based on user state assessment

ABSTRACT

A method and system are provided that prioritize the output of an image triage that is based on rapid serial visual presentation. User responses and estimates of the effectiveness with which each image is likely to have been processed by a user are employed for post triage image prioritization of potential targets. Images associated with a user response, processed during optimal user states, are assigned the highest priority for post triage examination, as targets are likely. Images without a user response that are processed during optimal user states are assigned the lowest priority, as these are unlikely to contain targets. Images with a user response that are processed during suboptimal states are assigned a medium priority, as these are likely to contain a high number of false positives. Images without a user response, processed during suboptimal user states are flagged for reprocessing as these may contain targets that the user may not have detected.

The present application is related to U.S. Patent ApplicationPCT/US07/01377, entitled METHOD AND SYSTEM FOR USER SENSITIVE PACINGDURING RAPID SERIAL VISUAL PRESENTATION, which is incorporated herein byreference.

BACKGROUND TECHNOLOGY

Rapid serial visual presentation (RSVP) allows high volumes of imageryto be searched efficiently by presenting images at rates of tens orhundreds of milliseconds per image. Hence, RSVP can be used to conduct atriage of high volumes of imagery. A triage generally refers to therapid identification, sorting, and stratification of images by theirlikelihood of containing critical information. Images of interest, ortargets, can be tagged either through the press of a button, or throughneurophysiological signals associated with target detection. Forexample, it is possible to use an evoked response potential (ERP) inelectroencephalogram (EEG) signals as a target detection cue. An ERP isa brief change in the brain's electrical potential in response tocritical events in the environment.

The output of the triage process may be a prioritized list of images,with images that elicited a user response being assigned the highestpriority, while those images without a user response being assigned alower priority. Unfortunately, if the output of the triage process isprioritized solely on the basis of ERP, the triage is likely to beinefficient or inadequate. This is because the ERP is only a validindicator of the presence or absence of a potential target, if imagesare processed effectively.

While RSVP can be effective when a user is attentive and appropriatelyfixated on the display screen, a variety of physical and cognitivestates can lead to missed targets. The current approaches to RSVP ignorethese physical and cognitive states.

In the intelligence community, the ability to extract useful informationfrom the terabytes of intelligence imagery gathered every day is limitedby the number of image analysts available and the slow pace of themanual triage process. Surveillance assets routinely capture informationthat could contribute to tactical successes, and minimize casualtiesamong military personnel and civilians. However, the ability to use thisdata effectively is contingent on rapid and accurate screening ofintelligence imagery. Unfortunately, with the limited number of imageanalysts available, and the time it takes to process each image, vastnumbers of images are not examined properly.

BRIEF DESCRIPTION OF THE DRAWINGS

Features of the present invention will become apparent to those skilledin the art from the following description with reference to thedrawings. Understanding that the drawings depict only typicalembodiments of the invention and are not therefore to be consideredlimiting in scope, the invention will be described with additionalspecificity and detail through the use of the accompanying drawings, inwhich:

FIG. 1 is a block diagram flowchart showing a fusion detection approachfor classification of image data;

FIGS. 2A and 2B are schematic diagrams depicting triage processes, witha traditional RSVP approach shown in FIG. 2A, and a user sensitivepacing approach shown in FIG. 2B;

FIG. 3 is a chart showing a prioritization and categorization scheme forimage triage output; and

FIG. 4 is a schematic depiction of one embodiment of an image triagesystem.

DETAILED DESCRIPTION

In the following detailed description, embodiments are described insufficient detail to enable those skilled in the art to practice theinvention. It is to be understood that other embodiments may be utilizedwithout departing from the scope of the present invention. The followingdetailed description is, therefore, not to be taken in a limiting sense.

The present invention relates to a method and system that prioritizesthe output of an RSVP based image triage system by considering both userresponses and estimates of the effectiveness with which each image islikely to have been processed by users. In general, the method forprioritizing an output of an image triage includes monitoring a physicalor cognitive state of a user; assigning a first priority to a set of oneor more images associated with an optimal user state when a userresponse is detected; assigning a second priority to a set of one ormore images associated with a suboptimal user state when a user responseis detected; and assigning a third priority to a set of one or moreimages associated with an optimal user state when a user response is notdetected. A set of one or more images associated with a suboptimal userstate can be reexamined when a user response is not detected.

For example, images associated with a user response such as an evokedresponse potential, processed during optimal user states, are assignedthe highest priority for post triage examination as these are highlylikely to contain targets. Images without a user response that areprocessed during optimal user states are assigned the lowest priority,as these are unlikely to contain targets. Images with a user responsethat are processed during suboptimal states are assigned a mediumpriority, as these are likely to contain a high number of falsepositives. Images without a user response, processed during suboptimaluser states are flagged for re-processing as these may contain targetsthat the user may not have detected.

Each image can be tagged with an estimate that can be used, inconjunction with user responses, for post triage image prioritization.The prioritization of triage output by considering the likelihood offalse positives and false negatives, which is based on user stateassessment, raises the overall efficiency and effectiveness of posttriage image analysis.

Estimates of the effectiveness with which a user is likely to haveprocessed images can be derived using a variety of sensors that measurecognitive and physical states associated with visual search. Cognitivestates such as attention levels and working memory load are known toaffect visual search. Attention levels and working memory load can beestimated using cardiac, EEG, or functional magnetic resonance imaging(fMRI) sensors. Additionally, physical states such as head orientation,eye blinks, eye position, eye scan patterns, and posture have an impacton visual search. These states can be identified using sensors such ashead trackers, body worn gyroscopes, eye trackers, and eye electrodes.The image triage system can flag images that were processed improperlydue to eye blinks, head movements, drops in attention, or a high workingmemory load.

The image triage system that utilizes the prioritization technique ofthe invention can include a variety of RSVP display modalities, so thatusers have a choice of formats appropriate for their domain. The imagetriage system can be modular in design, so that other various detectionapproaches can be easily integrated into the system. The image triagesystem can support time synchronization and logging for all sensor dataand software components. A signal processing module can be used tostreamline the flow of data and minimize redundant operations.

By detecting and mitigating suboptimal user states, and thereby raisingthe effectiveness of the human analyst, the present invention allowsintelligence analysts to focus more of their time and effort onanalyzing images that are most likely to contain targets. The net resultwill be more accurate and timely intelligence information for militaryand political decision makers.

Various aspects of the present invention are described in further detailin the following sections.

Real-Time Detection

The low signal to noise ratio inherent in evoked response potential(ERP) signals presents a difficult challenge for reliable ERP detection.Traditionally, ERP signals are averaged across repeated presentations ofstimuli to separate ERP signals from background EEG. Such a solution isimpractical in application contexts with real time requirements. Thepresent approach integrates information spatially across electrode sitesand examines EEG activity within a short time window around stimuluspresentation. This allows the construction of discriminant functionsthat help distinguish between an ERP signal and background EEG withinthese temporally restricted windows.

The present detection approach uses a complementary set of fast,single-trial techniques to detect ERP reliably. These include linearprojection, a nonlinear matched filter, and estimation of time frequencydistributions using wavelets. Each of these approaches contribute to alarge pool of features that help discriminate between the presence andabsence of ERPs. Classification of ERPs are based on a fusion of thesefeatures to a reduced dimensionality representation that maximizes theratio of relevant discriminative information content to the irrelevantdistractive content. A committee of state-of-the-art classifiers withminimal offline training requirements can be employed to obtain thefinal detection decision.

A. Linear Approach

The linear ERP approach relies on the assumption that measured EEGsignals x(t) are a linear combination of distributed source activitys(t) and zero-mean white Gaussian measurement noise n(t), which isdefined completely by its second-order statistics: x(t)=As(t)+n(t).Consequently, the optimal ERP detection strategy under this assumptionis to determine optimal linear projections of sensor measurementdiscriminability. For example, in the case of one-dimensionalprojections, this corresponds to projecting the sensor vector onto aweight vector w, y(t)=w^(T)x(t)+b. The linear projections can beoptimized using the traditional Fiser linear discriminant analysis (LDA)criterion, (m₀-m₁)²/(σ₀ ²+σ₁ ²), or alternatively, using the logisticregression technique that assumes the conditional class probabilitygiven the projection will follow a logistic model: P(c|y)=1/(1+e^(y)),which is consistent with the Gaussianity assumption.

While these techniques can provide acceptable levels of performance insome situations, they are restricted in their ability to accommodate anynonlinear amplitude and temporal distortions that the ERP waveforms mayexhibit from trial to trial even within the same session with the samesubject. Such deviations can render the linearity and Gaussianityassumptions invalid, thus leading to suboptimal detection performance.

B. Nonlinear Matched Filter

The nonlinear matched filter for ERP detection relies on kernel basedprojection techniques that are used for machine learning. Kernel basedtransformations provide a way to convert nonlinear solutions into linearsolutions via a projection into a higher dimensional space. Thisapproach uses an information theoretic concept called mutual information(MI) to identify optimal parameters for the kernel function used in theprojection. The MI is an objective measure of the dependency ornonlinear correlation between two or more random quantities. Thissuggests that the larger the MI between a set of EEG-based features andthe class labels (e.g., background EEG vs. ERP), the better the expectedclassification accuracy. Hence, the design of a nonlinear projectionthat maximizes the mutual information between the EEG projection andclass labels can be used to create a filter that optimally separates ERPfrom background EEG activity.

The nonparametric techniques used to design the nonlinear matched filterfor ERP detection make minimal assumptions regarding statistics of theunderlying data. Additionally, these techniques are mathematicallyproven to demonstrate very good small-sample size accuracy and fastconvergence to the true data statistics as more samples are used by theestimator.

C. Time Frequency Distribution

The ERP waveforms occur at varying times following stimuli presentation,thus it is imperative to take into account the temporal fluctuations inthe frequency distribution EEG signals. Since the ERP waveforms aretransient in nature, it is important to discover features that capturediscriminatory EEG features locally in time. The time frequencydistribution (TFD) is constructed using wavelets and can be estimatedusing Morlet Wavelet decomposition. This decomposition provides anoptimal time-frequency resolution for TFD estimation when wavelets arechosen appropriately. The squared-norm of each wavelet filter outputprovides an estimate of the energy in the time interval and frequencyband corresponding to the specific wavelet, and these features areobtained for each EEG site. The spatio-temporal distribution of EEGenergy at traditional bands (alpha, beta, theta, etc.) are utilized asthe features for discrimination.

While TFDs computed using wavelets provide excellent temporal andfrequency resolution, in the interest of computational efficiency andclassification accuracy, it is still necessary to separatediscriminatory TFD features from non-discriminatory features. To thisend, a variant of the best-bases algorithm can be employed. This methodexpands the EEG signal into orthonormal bases using wavelet packets overa dyadic grid (binary tree). This representation allows for efficientcompression of the TFD information, if this tree is pruned using anentropy criterion. The pruning can be based on the MI techniquesdescribed earlier. This helps to determine the most efficient anddiscriminatory sparse signal representation. Once discriminatoryfeatures have been identified using MI techniques, the relevant TFDfeatures can be used in conjunction with the linear and nonlinearprojection approaches described earlier as the basis for classification.

D. Fusion Detection Approach

While each of the above real-time ERP detection approaches can be usedindependently of the others in the image triage system, it is alsopossible to use these approaches jointly in various configurations oftwo or all three approaches. Such a fusion detection approach forclassification is illustrated in the block diagram flowchart of FIG. 1,and relies on extracting the most informative features provided by eachERP detection technique. The three ERP detection approaches describedabove can be employed to create a pool of potentially useful statisticsfor ERP detection. As shown in FIG. 1, these include linear projections110 and nonlinear matched filter projections 112 of raw data, andwavelet-based time frequency distributions 114 of power in EEG signals.This initial pool of features 120 can be diversified to capture a broadrange of critical features present in the EEG signals. These featuresare then evaluated for their optimally discriminatory value using MIbased feature ranking algorithms. Feature extraction using MI techniquescan be carried out in conjunction with the system calibration process.Once the optimal feature subset is identified, real-time ERPclassification will be extremely efficient.

Basing real-time classification on a feature subset that optimizesdiscriminability among classes allows the ERP decision to be made usingclassification techniques that require minimal or no online trainingrequirements. Examples of suitable classification techniques include KNearest Neighbor (KNN), Parzen Windows, and Gaussian Mixture Models(GMM). The KNN and Parzen techniques require no training, while GMMmodels the underlying data distributions very quickly. These alternativetechniques can be used in the context of a committee of classifiers 130,as shown in FIG. 1. The final ERP classification 140 chosen is the modaloutput decision of the three classifiers.

Each of the ERP detection approaches described above requires similaramounts of training data. The present detection system is calibratedusing a set of training images with known truth labels at the beginningof every session. Once calibrated, the ERP detection system providessingle trial ERP decisions well within real time constraints.

While the ERP detection system can be initially calibrated in the mannerdescribed above, it is also possible to implement an option for the ERPdetection system to adapt online during actual use. This is done tocompensate for long term EEG non-stationarity. Adaptation can beaccomplished by interleaving previously labeled data into the imagesequence and tuning the system based on associated EEG responses.

Cognitive State Estimation

The overall classification accuracy of an ERP-based triage system hingeson the effectiveness with which the user will be able to process images.A human analyst, engaged in the process of scanning images, adopts therole of a target sensor. The alert human analyst's target detectionabilities far exceed that of any mechanical sensor. However, unlikemechanical sensors that can remain perfectly fixated on the stimuli atevery instant, humans exhibit a great deal of variability in theircognitive and physical state over time. Humans blink, get tired, loseattention, and may be drawn to divert their eyes momentarily to otherelements in a room. Presenting images at rates of about 20 ms to about100 ms without any consideration for a user's evolving state is likelyto lead to missed targets.

It is important to have ways to determine whether an analyst is likelyto have perceived targets and to take steps to mitigate compromisedhuman performance that result in errors. Such errors fall into two broadcategories, false positives and false negatives.

False positives occur when the system classifies an image without atarget as one with a target. The cost of false positives is largelyrealized in the form of inefficiency—the incidence of false positivesforces analysts to weed out several irrelevant images among images ofinterest.

False negatives (or misses) occur when images containing a target areincorrectly classified as lacking a target. In many operationalcontexts, the cost of false negatives may be substantially higher thanthe cost associated with false positives. If potential targets goundetected in the triage process, vital information may fail to bescrutinized by analysts. The cost of omission may range from the loss ofa tactical or strategic advantage, to the loss of lives.

Sub-optimal cognitive and physical states can be detected by varioussensors and classifiers that have been adapted to detect sub-optimaluser states and invoke mitigation strategies. The sub-optimal statesdetected and mitigated by the present system are described as follows.

A. User Attention

Maintaining sustained attention over time is a difficult task for mosthumans. Researchers have noted that subjects performing visual searchtasks over long periods of time encounter a vigilance decrement thatleads to slower reaction times and increased error rates. However, ithas been found that momentary fluctuations in attention levels can beestimated using EEG. For instance, it has been noted that increases inspectral power at 4 Hz and 14 Hz in midline sites accompany periods oflow alertness, and that these changes can be used to reliably classifyperiods of low attention. Cognitive state classifiers can be used in thepresent system to detect inappropriate levels of attention and adapt thesystem appropriately to compensate for potential performance decrements.

B. Working Memory

Research suggests that working memory load constrains performance invisual search tasks. Working memory mechanisms play a role in helpingindividuals distinguish between currently task-relevant and irrelevantstimuli. This is of particular relevance to the task domain of theintelligence image analyst where users will be dealing with varied andambiguous targets and distractors. Reducing availability of workingmemory in the context of sustained attention tasks can compromiseperformance on target detection tasks.

The present approach incorporates EEG-based classifiers to assess auser's working memory load. Research indicates that increases in workingmemory demands contribute to an increase in frontal midline theta and adecrease in parietal alpha. Classifiers are employed to assess a user'sevolving cognitive load. As working memory loads approach levels thatcould negatively affect performance, the system can adapt to mitigatethe risk associated with suboptimal performance.

C. Gross Eye Activity

At the 20 ms to 100 ms rate of presentation common in RSVP tasks, avariety of normal eye activities can prevent images from being analyzed.Eye blinks occur at an average frequency of one every six seconds andlast for an average duration of 80 to 100 milliseconds. Images presentedin conjunction with eye blinks are unlikely to be processed veryeffectively. Over the span of sessions lasting tens of minutes, eyeblinks could result in hundreds of images going by without anappropriate degree of visual processing by the analyst. Unfortunately,eye blinks are only one example of eye activity that can impactperformance. Large eye saccades, such as a momentary glance away fromthe screen to relieve eye strain or to attend to an external event couldcause images to go by without being assessed by the analyst.

Fortunately, many of these eye events can be detected using EEG eyeelectrodes and unobtrusive desktop eye trackers. Specific ways in whichthe present system can use information from these sensors to mitigatethe detrimental impact of common activity will be discussed hereafter.

D. Head Orientation

Like the information provided by sensors assessing eye activity,assessments of head orientation provide a way to determine whether thesubject is likely to perceive information presented on the screen. It isnatural for users to vary the orientation of their head over the courseof a sustained period of time. Users may have to refer to other sourcesof information, monitor events in the surrounding environment, and varyposition to relieve fatigue. In the context of a user reviewing imagesin RSVP contexts, these routine actions could lead to missed images.Many of these actions occur frequently and may not even be somethingusers are consciously aware of as they work on tasks. Hence, users maynot think to pause the presentation. Small unobtrusive head trackers canbe employed to detect head positions that could compromise performance.Specific ways in which the present system can use information from thesesensors to mitigate the detrimental impact of common activity will bediscussed hereafter.

Human Performance Optimization

The image triage system described herein leverages the unsurpassedstrengths of the human analyst in target detection tasks, whileminimizing human limitations. While it may be tempting to think of theanalyst as being capable of fixating and attending to images at highrates, over sustained periods of time, the human is a complex systemwhose performance waxes and wanes. The present system allows users toprocess images at their maximum possible capacity, whatever that mightbe, at any instant. Towards this end, three mitigation techniques can beemployed in the system: user alerts, user sensitive pacing, and usersensitive prioritization, which are described as follows.

A. User Alerts

One technique that the triage system can employ to engage the useroptimally is to alert the user when certain suboptimal states aredetected. Aural and visual cues serve to notify users that they may notbe processing images effectively. Such aural and visual alerts arereserved for sustained lapses only. That is, for lapses that extend fortens of seconds or minutes.

B. User Sensitive Pacing

Many of the cognitive and physical states that can compromiseperformance are frequent and brief. Eye blinks, gross eye movements, andmomentary lapses in attention are likely to occur frequently and bebeyond the control or conscious awareness of the user. Reminders inresponse to these frequent events could be frustrating and distractingto the user. A technique called user sensitive pacing is used tomitigate these events.

The user sensitive pacing technique optimizes the flow of images to theanalyst in real time. This allows for images to be presented at a pacethat is as high as an analyst can effectively handle at any instant.This is done by using the cognitive and physical state sensingtechniques described earlier. FIGS. 2A and 2B are schematic diagramsdepicting triage processes, with a traditional RSVP approach 210 shownin FIG. 2A, and a user sensitive pacing approach 250 shown in FIG. 2B.

FIG. 2A shows the consequences of suboptimal states on triageeffectiveness for the traditional RSVP approach 210. During analysis ofa set of images 212, an eye blink 214 or a fall in attention 216 resultsin one or more missed images 220, which are never further considered.This results in a less than effective triage process, which can resultin detrimental consequences such as false negatives and false positives.

In the user sensitive pacing approach 250 shown in FIG. 2B, the physicaland cognitive states described previously are taken into considerationfor distinct pacing interventions. For instance, during analysis of aset of images 252, user sensitive pacing accommodates for eye blink 254.Once a blink is detected, the image index is set to the imageimmediately preceding the blink. The missed image sequences 260 arecached in memory so that fast re-indexing can occur without perceivableinterruption to the user. The image frames affected by the eye blinkscan then be re-visited when desired. Additionally, gross eye saccadesthat lead to fixation away from the image screen will cause thepresentation of images to pause. The current image index is then movedto the image that occurred just prior to the beginning of the saccade.The presentation of images will resume as soon as the eyes return toappropriate fixation regions. Head movements can be treated in a waysimilar to eye saccades.

In the user sensitive pacing approach, attention and working memoryassessments serve to control the rate of image presentation. As shown inFIG. 2B, if there is a fall in attention 270, then an image presentationrate 272 is adjusted appropriately. Thus, as a user's attention levelwanes or working memory load begins to increase, the pace of imagepresentation can be reduced. For example, at low working memory load andhigh attention levels, images can be presented at rates of about 20-100ms. At medium levels of attention and working memory, the presentationrate can be reduced to an image at about every 200 ms. At a high workingmemory and low attention levels, image presentation may be temporarilyreduced to one at about every 300 ms. These presentation rates are forillustration only as other rates might be more appropriate in aparticular analysis context.

Image sensitive pacing allows users to interact in a natural manner withthe triage system. Users will not have to deal with unreasonable demandson their physical or cognitive state. Instead, the system continuallyadapts to the user. User sensitive pacing allows the system to leverageuser strengths while minimizing the impact of normal humanvulnerabilities in RSVP tasks.

The user sensitive pacing technique is described further in copendingU.S. Patent Application PCT/US07/01377, entitled METHOD AND SYSTEM FORUSER SENSITIVE PACING DURING RAPID SERIAL VISUAL PRESENTATION, which isincorporated herein by reference.

C. User Sensitive Prioritization

While user sensitive pacing is used to maximize the likelihood thatimages are processed appropriately during analysis, user sensitiveprioritization is used to organize the output of the triage processafter analysis. User sensitive prioritization relies on ERP-based targetclassification, in combination with assessments of cognitive andphysical state, to categorize and prioritize scanned images. Imagecategorization is done in terms of the likelihood of containing targets.

For example, images that elicit an ERP and occur during nominal(optimal) user states are classified as likely true positives. Imagesthat do not elicit an ERP and occur during nominal user states areclassified as likely true negatives. Images that elicit an ERP and occurduring sub-optimal user states are classified as potential falsepositives. Images that do not elicit an ERP and occur during sub-optimaluser states are classified as potential false negatives.

Once the outputs of the ERP based triage are categorized, the outputscan be prioritized for closer review by the analyst. FIG. 3 is a chartshowing the categorization and prioritization of triage output, andsummarizes one possible prioritization scheme that can be used by thepresent triage system. The ideal prioritization scheme will vary bycontext and can be specified by the user.

As indicated by FIG. 3, images that are viewed during periods of optimaluser state, and elicit an ERP, are assigned the highest priority forpost triage review as targets are likely. Images that elicit an ERP, butoccur during suboptimal user states, are assigned a lower (medium)priority as targets are less likely. Images that do not produce an ERP,but occur during optimal user states are assigned the lowest priority,as the likelihood of targets being present in these images will be quitelow. Images that do not produce an ERP and occur during suboptimal userstates are labeled for rescanning as the target status is unknown.

User sensitive prioritization provides a way to distinguish between truenegatives and false negatives. Without independent assessment of userstate as described herein, such disambiguation becomes impossible.

Image Triage System

FIG. 4 is a schematic depiction of one embodiment of an image triagesystem 400 according to the present invention, which has modular systemcomponents. Various software and hardware components are used to providea real-time triage system. The modular architecture design allows easyintegration of new software or hardware components if desired. Thetriage system 400 streamlines the handling of data, eliminates redundantprocessing, supports logging, provides precise time synchronization, andsupports RSVP display formats relevant to various types of imagery. Asshown in FIG. 4, the image triage system 400 generally includes adisplay module 410, a detection module 420, and a sensor module 430, allof which are in operative communication with each other. A base stationcomputer can be used to control the triage system and can communicatewith the system via a wireless network or Ethernet connection.

The display module 410 provides a means for managing images that areshown to a user. The image triage system 400 integrates an RSVPinterface display 412 and an image database 414, which are part ofdisplay module 410. The interface display 412 can use a variety ofdifferent RSVP presentation formats that are shown on a display screen416 to a user. Given the fact that analysts deal with information from abroad range of information sources, the present system provides theanalyst a choice of interfaces. Examples of interface display modalitiesthat can be provided by the system are keyhole, carousel, and floatingdisplays.

Keyhole RSVP displays present images in a slideshow format, with allimages being displayed at the same location. This modality may be mostsuited for static imagery. Carousel RSVP displays present several imageson the screen at the same time. Images start from the left, displayed ina small format, and grow in size until they reach the top of the screen,and diminish in size as they move to the right. A variant of thecarousel display may be appropriate for processing broad area imagery asit provides a sense for surrounding spatial context. Floating RSVPdisplays extend a series of frames into a 3-D trail. The frame in theforeground is analogous to a vehicle windshield. Frames in the distancebegin approaching the user and fade away. Floating RSVP is particularlyeffective for detecting targets within video frames.

Display modalities like carousel and floating RSVP provide users with abroader sense for spatiotemporal dynamics of a scene than the keyholedisplay. In many application domains, the broader context provided bythese schemes may improve target detection relative to the narrowperspective provided by the keyhole display.

The image database 414 can include a variety of different image typessuch as static and broad area images, video clips, image chips, and thelike. Image chips are produced from a large image that is “cut up” intoa series of smaller images that are presented one after the other.

The detection module 420 provides a means for detecting ERP in a user.The detection module 420 employs an integrated real-time ERP featuredetection system 422 that can include one or more of the variouscomponents discussed previously, such as linear projection, nonlinearmatched filters, and estimations of time frequency distributions usingwavelets. A feature fusion system 424 can also be implemented to providefor complementary use of these techniques such as in the fusion approachdescribed above with respect to FIG. 1. This adds redundancy androbustness to the triage system and improves overall ERP detectionaccuracy. A cognitive classification system 426 is also part ofdetection module 420.

The sensor module 430 provides a means for monitoring the physical andcognitive state of an analyst user. The sensor module 430 can includevarious standard sensor components used to detect the state of theanalyst user. Such sensor components can include sensors for monitoringworking memory 432, user attention 434, eye activity 436, and headorientation 438.

For example, EEG data can be collected using the BioSemi Active Twosystem. This system has a 32 channel EEG cap and a set of eyeelectrodes. The eye electrodes provide information concerning eye blinksand eye saccades. The BioSemi system integrates an amplifier with anAg—AgCl electrode, which affords extremely low noise measurementswithout any skin preparation. Information about head orientation can beprovided by, for example, the InertiaCube. The InertiaCube providesorientation information about the head's pitch, roll, and yaw axes.Information from these sensors can be processed on a standard personalcomputer (PC). User gaze can be tracked with an unobtrusive desk mountedtracking system that provides face, eye, eyelid and gaze tracking usinga completely non-contact, video-based sensor. The sensors can beconnected to the PC via a combination of USB ports, serial ports, orBluetooth wireless interfaces.

A signal processing module 440 provides a means for processing signalsfrom the sensor module 430 prior to feature extraction andclassification by detection module 420. The signal processing module 440can incorporate one or more signal drift correction filters or bandpassfilters. For example, filters can be used that correct for DC drift ofsignals over time. Bandpass filters can be implemented to allow signalprocessing components to extract frequency bands of interest for furtheranalysis.

The signal processing module 440 can incorporate components to correctfor eye blink artifacts. Effective decontamination of eye activity isparticularly important for ERP classification. High amplitude noiseassociated with eye activity contributes to the overall challenge ofreliably detecting ERPs. An adaptive linear ocular filter that removeseye blink artifacts from EEG signals can be used and is available fromHoneywell.

The signal processing module 440 can also incorporate components toprovide power spectral density (PSD) estimates. Classifiers associatedwith attention and cognitive load, use estimates of spectral power atvarious frequency bands as input features for classification. Tominimize redundant operations, a single component can be used togenerate PSD estimates and propagate them to components that rely on PSDestimates as input features. The PSD of EEG signals can be estimatedusing the Welch method. The PSD process uses 1-second sliding windowswith 50% overlap. PSD estimates are integrated over five frequencybands: 4-8 Hz (theta), 8-12 Hz (alpha), 12-16 Hz (low beta), 16-30 Hz(high beta), and 30-44 Hz (gamma). These bands sampled every 0.1 secondscan be used as the basic input features for cognitive classification.The particular selection of the frequency bands is based onwell-established interpretations of EEG signals in prior cognitive andclinical contexts.

During operation of image triage system 400, a user 450 looks at displayscreen 416 that is provided with a set of images for analysis from imagedatabase 414 of display module 410. The sensors worn by the user as partof sensor module 430 detect signals generated from the physical andcognitive state of the user. These signals are processed by signalprocessing module 440 for use by detection module 420 in detecting theERP in user 450. The images viewed by user 450 are time synchronized sothat they correspond with the detected ERP.

The detection module 420 and sensor module 430 communicate with ananalyst sensitive categorization/prioritization system 470 as shown inFIG. 4, which provides a means for assigning a various priorities tosets of one or more images associated with optimal or suboptimal userstates when a user response is detected or not detected. The analystsensitive prioritization system 470 can be implemented using variouscomputer hardware and software components. The triage output can be sentto a storage device 480 for post triage examination of images. Thestorage device 480 also provides a means for reexamining imagesassociated with a suboptimal user state when a user response is notdetected.

In the analyst sensitive prioritization system 470, images associatedwith a user response, processed during optimal user states, are assignedthe highest priority for post triage examination. Images without a userresponse that are processed during optimal user states are assigned thelowest priority. Images with a user response that are processed duringsuboptimal states are assigned a medium priority. Images without a userresponse, processed during suboptimal user states are flagged forre-processing.

Although not required, the image triage system 400 can also implement ananalyst sensitive pacing system 484 and/or an analyst alert system 488,which are in communication with sensor module 430 and display module 410as depicted in FIG. 4.

Instructions for carrying out the various methods, process tasks,calculations, control functions, and the generation of signals and otherdata used in the operation of the system are implemented, in someembodiments, in software programs, firmware or computer readableinstructions. These instructions are typically stored on any appropriatemedium used for storage of computer readable instructions such as floppydisks, conventional hard disks, CD-ROM, flash memory ROM, nonvolatileROM, RAM, and other like medium.

The present invention may be embodied in other specific forms withoutdeparting from its essential characteristics. The described embodimentsare to be considered in all respects only as illustrative and notrestrictive. The scope of the invention is therefore indicated by theappended claims rather than by the foregoing description. All changesthat come within the meaning and range of equivalency of the claims areto be embraced within their scope.

1. A method for prioritizing an output of an image triage, comprising:displaying images to a user on a display screen; using one or moresensors to monitor a physical state or cognitive state of the user whiledisplaying the images to the user; detecting an evoked responsepotential (ERP) in the user while displaying the images to the user;classifying the user as being in one of two distinct user states basedon the monitored physical state or cognitive state of the user, the twodistinct user states being an optimal user state or a suboptimal userstate; and in a user sensitive prioritization system: assigning a firstpriority to a set of one or more of the images when the user isclassified as being in the optimal user state and an ERP is detected;assigning a second priority to a set of one or more of the images whenthe user is classified as being in the suboptimal user state and an ERPis detected; and assigning a third priority to a set of one or more ofthe images when the user is classified as being in the optimal userstate and an ERP is not detected.
 2. The method of claim 1, furthercomprising reexamining a set of one or more of the images when the useris classified as being in the suboptimal user state and an ERP isdetected.
 3. The method of claim 1, wherein the image triage comprises arapid serial visual presentation.
 4. The method of claim 1, wherein thephysical state or cognitive state of the user is monitored by one ormore sensors.
 5. The method of claim 4, wherein the physical state ofthe user comprises one or more of head orientation, eye blinks, eyeposition, eye scan patterns, or body posture.
 6. The method of claim 4,wherein the cognitive state of the user comprises one or more ofattention level or working memory load.
 7. The method of claim 1,wherein the first priority is higher than the second priority, and thesecond priority is higher than the third priority.
 8. The method ofclaim 1, wherein the user response is an evoked response potential. 9.The method of claim 8, wherein the evoked response potential is detectedby one or more techniques comprising a linear projection, a nonlinearmatched filter, or a wavelet-based time frequency distribution.
 10. Animage triage system, comprising: a display module configured to displaya plurality of images to a user; a detection module in operativecommunication with the display module and configured to detect an evokedresponse potential (ERP) in the user; a sensor module in operativecommunication with the display module and the detection module andconfigured to sense a physical state or cognitive state of the userwhile displaying the images to the user and, based on the sensedphysical state or cognitive state, classify the user as being in one oftwo distinct user states, the two distinct user states being an optimaluser state or a suboptimal user state; a signal processing module inoperative communication with the sensor module and the detection moduleand configured to process signals therefrom; and a user sensitiveprioritization system in operative communication with the detectionmodule and the sensor module; wherein the user sensitive prioritizationsystem assigns various priorities to sets of one or more of theplurality of images based upon (i) whether the user is classified asbeing in the optimal or suboptimal user state and (ii) whether an ERP isdetected or not detected.
 11. The system of claim 10, wherein thedisplay module comprises a rapid serial visual presentation display, andan image database.
 12. The system of claim 10, wherein the evokedresponse potential feature detection system comprises a linearprojection, a nonlinear matched filter, a wavelet-based time frequencydistribution, or combinations thereof.
 13. The system of claim 10,wherein the detection module comprises a feature fusion system.
 14. Thesystem of claim 10, wherein the detection module comprises a cognitiveclassification system.
 15. The system of claim 10, wherein the sensormodule comprises one or more of a working memory sensor, a userattention sensor, an eye activity sensor, or a head orientation sensor.16. The system of claim 10, wherein the signal processing modulecomprises one or more signal drift correction filters, bandpass filters,linear ocular filters, or components to provide power spectral densityestimates.
 17. The system of claim 10, further comprising an outputstorage device for post triage examination of the plurality of images.18. The system of claim 10, further comprising one or more of a usersensitive pacing system or a user alert system, which are incommunication with the sensor module and the display module.